Project 1| Unsupervised Learning¶
The purpose of the case study is to classify a given silhouette as one of four different types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars. The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
In [1]:
%matplotlib inline
## Import libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_color_codes()
In [2]:
# load the dataset
sh_df = pd.read_csv('vehicle.csv')
sh_df.head()
Out[2]:
In [3]:
# Shape of the dataset
sh_df.shape
Out[3]:
1. Data pre-processing - Understand the data and treat missing values (Use box plot), outliers (5 points)¶
1a - Understand the data
1b - Find missing values
1c - Treat missing values
1d - Find outliers
1e - Treat outliers
1a - Understand the data¶
In [4]:
sh_df.info()
# Distribution of data types -
# dtypes: float64(14), int64(4), object(1)
In [5]:
sh_df.describe(include='all').transpose()
Out[5]:
In [7]:
# Finding Unique values
for i in sh_df.columns:
print(i ,':', sh_df[i].unique() )
In [8]:
pd.value_counts(sh_df['class']).plot(kind = 'bar')
plt.title('Bar Plot for Class variable')
plt.xlabel('Class')
plt.ylabel('Value Counts')
plt.show()
In [86]:
sh_df['class'].value_counts()
Out[86]:
1b - Find missing values¶
In [9]:
sh_df.isnull().sum()
Out[9]:
Since above technique did not show missing values we will loop through individual column
In [11]:
sh_df.isna().sum()
Out[11]:
Since above technique did not show missing values we will loop through individual column
1c - Treat missing values¶
In [13]:
# Both using isnull and isnan produced same missing values count.
# Fill the missing values with mean (as per obersations from the describe)
sh_df = sh_df.fillna(sh_df.mean())
In [14]:
sh_df.isnull().sum()
Out[14]:
In [15]:
## Verify after imputing
for col in sh_df.columns:
print('# Missing values for col \'{}\': {}'.format(col, sh_df[col].isna().sum()))
Univarivate Analysis, BI Varaite Analysis¶
In [16]:
sns.distplot(sh_df['compactness'])
Out[16]:
In [17]:
sns.distplot(sh_df['compactness'])
Out[17]:
In [18]:
sns.distplot(sh_df['compactness'])
Out[18]:
In [81]:
for i in sh_df.columns:
sh_df.groupby(i)['class'].value_counts().unstack().plot(kind = 'bar', stacked = True, figsize = (8,6))
In [72]:
sns.pairplot(sh_df)
Out[72]:
In [51]:
sns.pairplot(data=sh_df, diag_kind='kde', hue='class')
Out[51]:
In [87]:
sns.set()
sns.pairplot(sh_df, size = 2.0)
plt.show()
In [19]:
sh_df.corr()
Out[19]:
In [20]:
plt.subplots(figsize=(10,8))
sns.heatmap(sh_df.corr())
Out[20]:
Following variables are highly correlated (positively) : scaled_variance, scaled_variance.1, scatter_ratio, pr.axis_rectangulaity Following variable are highly negatively correlated: elongatedness, compactness, circularity, distance_circularity, radius_ratio, scatter_ratio, pr.axis_rectangularity, max.length_rectangularity, scaled_variance, scaled_variance.1, scaled_radius_of_gyration The variables, skewness_about.2, hollows_ratioo and scaled_radius_of_gyration.1 are also highly negatively correlated
From above it is clear the following features have outlier - radius_ratio, pr.axis_aspect_ratio, max.length_aspect_ratio, scaled_variance, scaled_variance.1, scaled_radius_of_gyration.1, skewness_about, skewness_about.1
1d - Find outliers¶
In [21]:
from itertools import chain
numeric_cols = sh_df.select_dtypes(include=['float64', 'int64']).columns
outlier_rec = []
# For each predictors find outliers using mathematical function.
def find_outlier(df_in, col_name, verbose=False):
q25 = df_in[col_name].quantile(0.25)
q75 = df_in[col_name].quantile(0.75)
iqr = q75-q25 #Interquartile range
lower, upper = q25-1.5*iqr, q75+1.5*iqr
outliers_df = df_in[(df_in[col_name] < lower) | (df_in[col_name] > upper)]
outliers_removed_df = df_in[(df_in[col_name] >= lower) & (df_in[col_name] <= upper)]
if(verbose):
print('# Number of outliers / non-outliers for column \'{}\': {} /{}'.format(
col_name, outliers_df.shape[0], outliers_removed_df.shape[0]))
return outliers_df.index.tolist();
for feature in numeric_cols:
outlier_rec.append(find_outlier(sh_df, feature, True))
outlier_rec = list(chain.from_iterable(outlier_rec))
outlier_rec = list(set(outlier_rec))
outlier_rec.sort()
print('# Total outliers in the dataset: {}'.format(len(outlier_rec)))
print(outlier_rec)
In [22]:
plt.figure(figsize=(15,10))
labels = sh_df.columns
sns.set(style='whitegrid')
sbplot = sns.boxplot(data=sh_df)
sbplot.set_xticklabels(labels=labels, rotation=45)
Out[22]:
In [23]:
fig, ax = plt.subplots(4,5, figsize=(22,20))
sns.boxplot(sh_df['compactness'], ax=ax[0,0])
sns.boxplot(sh_df['circularity'], ax=ax[0,1])
sns.boxplot(sh_df['distance_circularity'], ax=ax[0,2])
sns.boxplot(sh_df['radius_ratio'], ax=ax[0,3])
sns.boxplot(sh_df['pr.axis_aspect_ratio'], ax=ax[0,4])
sns.boxplot(sh_df['max.length_aspect_ratio'], ax=ax[1,0])
sns.boxplot(sh_df['scatter_ratio'], ax=ax[1,1])
sns.boxplot(sh_df['elongatedness'], ax=ax[1,2])
sns.boxplot(sh_df['pr.axis_rectangularity'], ax=ax[1,3])
sns.boxplot(sh_df['max.length_rectangularity'], ax=ax[1,4])
sns.boxplot(sh_df['scaled_variance'], ax=ax[2,0])
sns.boxplot(sh_df['scaled_radius_of_gyration'], ax=ax[2,1])
sns.boxplot(sh_df['scaled_radius_of_gyration.1'], ax=ax[2,2])
sns.boxplot(sh_df['skewness_about'], ax=ax[2,3])
sns.boxplot(sh_df['skewness_about.1'], ax=ax[2,4])
sns.boxplot(sh_df['skewness_about.2'], ax=ax[3,0])
sns.boxplot(sh_df['hollows_ratio'], ax=ax[3,1])
Out[23]:
From above it is clear the following features have outlier - radius_ratio, pr.axis_aspect_ratio, max.length_aspect_ratio, scaled_variance, scaled_variance.1, scaled_radius_of_gyration.1, skewness_about, skewness_about.
1e - Treat outliers¶
In [95]:
pd.crosstab(sh_df['radius_ratio'], sh_df['class'])
Out[95]:
In [96]:
sh_df['radius_ratio'] = np.where(sh_df['radius_ratio']>=300, 252, sh_df['radius_ratio'])
In [97]:
pd.crosstab(sh_df['radius_ratio'], sh_df['class'])
Out[97]:
In [98]:
pd.crosstab(sh_df['pr.axis_aspect_ratio'], sh_df['class'])
Out[98]:
In [99]:
sh_df['pr.axis_aspect_ratio'] = np.where(sh_df['pr.axis_aspect_ratio']>=76, 75, sh_df['pr.axis_aspect_ratio'])
In [100]:
pd.crosstab(sh_df['pr.axis_aspect_ratio'], sh_df['class'])
Out[100]:
In [101]:
pd.crosstab(sh_df['max.length_aspect_ratio'], sh_df['class'])
Out[101]:
In [102]:
sh_df['max.length_aspect_ratio'] = np.where(sh_df['max.length_aspect_ratio']>=13, 12, sh_df['max.length_aspect_ratio'])
In [103]:
pd.crosstab(sh_df['max.length_aspect_ratio'], sh_df['class'])
Out[103]:
In [104]:
pd.crosstab(sh_df['scaled_variance'], sh_df['class'])
Out[104]:
In [105]:
sh_df['scaled_variance'] = np.where(sh_df['scaled_variance']>=300, 288, sh_df['scaled_variance'])
In [106]:
pd.crosstab(sh_df['scaled_variance'], sh_df['class'])
Out[106]:
In [107]:
pd.crosstab(sh_df['skewness_about'], sh_df['class'])
Out[107]:
In [108]:
sh_df['skewness_about'] = np.where(sh_df['skewness_about']>=20, 19, sh_df['skewness_about'])
In [109]:
pd.crosstab(sh_df['skewness_about'], sh_df['class'])
Out[109]:
In [110]:
pd.crosstab(sh_df['scaled_radius_of_gyration'], sh_df['class'])
Out[110]:
In [115]:
sh_df['scaled_radius_of_gyration'] = np.where(sh_df['scaled_radius_of_gyration']>=200, 210, sh_df['scaled_radius_of_gyration'])
In [117]:
pd.crosstab(sh_df['scaled_radius_of_gyration'], sh_df['class'])
Out[117]:
In [113]:
# After Treating the outliers
plt.figure(figsize=(15,10))
labels = sh_df.columns
sns.set(style='whitegrid')
sbplot = sns.boxplot(data=sh_df)
sbplot.set_xticklabels(labels=labels, rotation=45)
Out[113]:
In [114]:
print('# Shape of dataset before removing outliers:{}'.format(sh_df.shape[0]))
## Treating ouliers. Action to take - drop the rows
# sh_df.drop(sh_df.index[outlier_rec], inplace=True)
print('# Shape of dataset before after outliers:{}'.format(sh_df.shape[0]))
2. Understanding the attributes - Find relationship between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why (2.5 points)¶
Two appraoch to find relationship between different attributes (Independent variable)¶¶
1) Using correlation matrix and plot in heatmap to see visually
2) Using variance inflation factor (VIF)
In [42]:
# Create correlation matrix
corr_matrix = sh_df.corr().abs()
# Select upper triangle of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))
# Find index of feature columns with correlation greater than 0.95
to_drop = [column for column in upper.columns if any(upper[column] > 0.95)]
to_drop
Out[42]:
In [43]:
plt.figure(figsize=(20,12))
sns.heatmap(corr_matrix, annot=True)
Out[43]:
In [44]:
# Standardize the dataset
from sklearn.preprocessing import StandardScaler
# Standardize the feature matrix
feature_space = pd.DataFrame(sh_df, columns=numeric_cols)
std_sh = StandardScaler().fit_transform(feature_space)
std_sh_df = pd.DataFrame(std_sh, columns=numeric_cols)
In [45]:
from statsmodels.stats.outliers_influence import variance_inflation_factor
predictor_variables = numeric_cols
threshold = 10
for i in np.arange(0,len(predictor_variables)):
vif = [variance_inflation_factor(std_sh_df[predictor_variables].values, j) for j in range(std_sh_df[predictor_variables].shape[1])]
maxindex = vif.index(max(vif))
if max(vif) > threshold:
#print ("VIF :", vif)
print('Eliminating \'' + std_sh_df[predictor_variables].columns[maxindex] + '\' at index: ' + str(maxindex))
#del predictor_variables[maxindex]
else:
break
3. Use PCA from scikit learn and elbow plot to find out reduced number of dimension (which covers more than 95% of the variance) - 10 points¶
In [46]:
C_df2 = sh_df.drop(columns=['class'])
In [47]:
from sklearn.decomposition import PCA
# Create a PCA that will retain 95% of the variance
pca_model = PCA()
X_test_reduced = pca_model.fit_transform(std_sh_df)
X_test_reduced.shape
Out[47]:
In [48]:
pca_model.components_
Out[48]:
In [49]:
e_variance = pca_model.explained_variance_
e_variance
Out[49]:
In [50]:
e_variance_ratio = pca_model.explained_variance_ratio_
e_variance_ratio
Out[50]:
In [51]:
np.cumsum(e_variance_ratio)
Out[51]:
In [52]:
#Plotting the Cumulative Summation of the Explained Variance
plt.figure()
plt.plot(np.cumsum(pca_model.explained_variance_ratio_))
plt.xlabel('Number of Components')
plt.ylabel('Variance (%)') #for each component
plt.title('Vehicle Dataset Explained Variance')
plt.show()
Based on the above cummulative sum, the first 3 columns covers 95% variance
Correlation between componenets and features¶
In [53]:
pca_df = pd.DataFrame(data=pca_model.components_, columns=C_df2.columns)
In [54]:
pca_df.head()
Out[54]:
In [55]:
pca_df.corr()
Out[55]:
In [56]:
sns.heatmap(pca_df)
Out[56]:
In [57]:
range_ = list(range(1,X_test_reduced.shape[1]+1))
plt.plot(range_, e_variance)
Out[57]:
In [58]:
cum_var_exp = np.cumsum(e_variance_ratio)
e_variance_ratio, cum_var_exp
Out[58]:
In [59]:
with plt.style.context('seaborn-whitegrid'):
plt.figure(figsize=(8, 6))
plt.bar(range(len(e_variance_ratio)), e_variance_ratio, alpha=0.5, align='center',
label='individual explained variance')
plt.step(range(len(e_variance_ratio)), cum_var_exp, where='mid',
label='cumulative explained variance')
plt.ylabel('Explained variance ratio')
plt.xlabel('Principal components')
plt.legend(loc='best')
plt.tight_layout()
In [60]:
# Repeating the above steps by removing the collinear columns
std_sh_drop_df = std_sh_df.drop(to_drop, axis=1)
print(std_sh_drop_df.shape)
std_sh_drop_df.head()
Out[60]:
In [68]:
pca_model = PCA(n_components=7)
X_test_reduced = pca_model.fit_transform(std_sh_drop_df)
In [69]:
pca_model.components_
Out[69]:
In [70]:
pca_model.explained_variance_
Out[70]:
In [71]:
pca_model.explained_variance_ratio_
Out[71]:
In [72]:
e_variance = pca_model.explained_variance_
range_ = list(range(1,std_sh_drop_df.shape[1]+1))
plt.plot(range_, e_variance)
Out[72]:
In [73]:
e_variance_ratio = pca_model.explained_variance_ratio_
cum_var_exp = np.cumsum(e_variance_ratio)
e_variance_ratio, cum_var_exp
Out[73]:
In [74]:
with plt.style.context('seaborn-whitegrid'):
plt.figure(figsize=(8, 6))
plt.bar(range(len(e_variance_ratio)), e_variance_ratio, alpha=0.5, align='center',
label='individual explained variance')
plt.step(range(len(e_variance_ratio)), cum_var_exp, where='mid',
label='cumulative explained variance')
plt.ylabel('Explained variance ratio')
plt.xlabel('Principal components')
plt.legend(loc='best')
plt.tight_layout()
In [75]:
corr_df = pd.DataFrame(pca_model.components_)
corr_df
Out[75]:
4. Use Support vector machines and use grid search (try C values - 0.01, 0.05, 0.5, 1 and kernel = linear, rbf) and find out the best hyper parameters and do cross validation to find the accuracy. (7.5 points)¶
SVM Approach - 1¶
In [76]:
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
In [77]:
parameter_candidates = [
{'C':[0.01, 0.05, 0.5, 1], 'kernel': ['linear']},
{'C':[0.01, 0.05, 0.5, 1], 'kernel': ['rbf']}
]
In [78]:
X = std_sh_df
y = sh_df['class']
# Create a classifier object with the classifier and parameter candidates
clf = GridSearchCV(estimator=SVC(), param_grid=parameter_candidates, cv = 5)
# Train the classifier on data1's feature and target data
clf.fit(X, y)
Out[78]:
In [79]:
# Let’s look at the accuracy score when we apply the model to the data1’s test data.
# View the accuracy score
print('Best score for data1:', clf.best_score_)
In [80]:
# Which parameters are the best? We can tell scikit-learn to display them:
# View the best parameters for the model found using grid search
print('Best C:',clf.best_estimator_.C)
print('Best Kernel:',clf.best_estimator_.kernel)
In [81]:
# Repeating the above steps dropping columns with coorelation > .95
X = std_sh_drop_df
y = sh_df['class']
# Create a classifier object with the classifier and parameter candidates
clf = GridSearchCV(estimator=SVC(), param_grid=parameter_candidates, cv = 5)
# Train the classifier on data1's feature and target data
clf.fit(X, y)
Out[81]:
In [82]:
# Let’s look at the accuracy score when we apply the model to the data1’s test data.
# View the accuracy score
print('Best score for data1:', clf.best_score_)
In [83]:
# Which parameters are the best? We can tell scikit-learn to display them:
# View the best parameters for the model found using grid search
print('Best C:',clf.best_estimator_.C)
print('Best Kernel:',clf.best_estimator_.kernel)
In [84]:
# Use SVC with redcued feature set fro PCA
X = X_test_reduced
y = sh_df['class']
# Create a classifier object with the classifier and parameter candidates
clf = GridSearchCV(estimator=SVC(), param_grid=parameter_candidates, cv = 5)
# Train the classifier on data1's feature and target data
clf.fit(X, y)
# View the accuracy score
print('Best score for data1:', clf.best_score_)
# Which parameters are the best? We can tell scikit-learn to display them:
# View the best parameters for the model found using grid search
print('Best C:',clf.best_estimator_.C)
print('Best Kernel:',clf.best_estimator_.kernel)
SVM Approach - 2¶
In [85]:
from sklearn.model_selection import train_test_split
# To calculate the accuracy score of the model
from sklearn.metrics import accuracy_score, confusion_matrix
target = sh_df["class"]
features = sh_df.drop(["class"], axis=1)
X_train, X_test, y_train, y_test = train_test_split(features, target, stratify=target, test_size = 0.2, random_state = 10)
In [118]:
print(X_train.shape)
print(X_test.shape)
In [119]:
print(y_train.shape)
print(y_test.shape)
In [120]:
# use from sklearn.svm import SVC
from sklearn.svm import SVC
# Building a Support Vector Machine on train data
svc_model = SVC(C= .1, kernel='linear', gamma= 1)
svc_model.fit(X_train, y_train)
prediction = svc_model.predict(X_test)
In [121]:
# check the accuracy on the training set
print(svc_model.score(X_train, y_train))
print(svc_model.score(X_test, y_test))
In [122]:
print("Confusion Matrix:\n",confusion_matrix(prediction,y_test))
In [123]:
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
scaled = scaler.fit(features)
In [124]:
data_scaled= scaler.transform(features)
In [125]:
data_scaled
Out[125]:
In [126]:
X_train, X_test, y_train, y_test = train_test_split(data_scaled, target, stratify=target, test_size = 0.2, random_state = 10)
In [127]:
# Building a Support Vector Machine on train data
svc_model = SVC(C= .1, kernel='linear', gamma= 1)
svc_model.fit(X_train, y_train)
prediction = svc_model.predict(X_test)
In [128]:
# check the accuracy on the training set
print(svc_model.score(X_train, y_train))
print(svc_model.score(X_test, y_test))
In [129]:
print("Confusion Matrix:\n",confusion_matrix(prediction,y_test))
In [130]:
# Building a Support Vector Machine on train data
svc_model = SVC(C= 1000, kernel='linear', gamma= 1)
svc_model.fit(X_train, y_train)
prediction = svc_model.predict(X_test)
In [131]:
# check the accuracy on the training set
print(svc_model.score(X_train, y_train))
print(svc_model.score(X_test, y_test))
In [132]:
print("Confusion Matrix:\n",confusion_matrix(prediction,y_test))
Inreasing C allowed us to improve the model
In [133]:
import multiprocessing
from sklearn.model_selection import GridSearchCV
In [134]:
param_grid = [ {
'kernel': ['linear', 'rbf'],
'C': [ 0.01, 0.05, 0.5, 1 ] } ]
In [135]:
gs = GridSearchCV(estimator=SVC(), param_grid=param_grid,scoring='accuracy', cv=10, n_jobs=multiprocessing.cpu_count())
In [136]:
gs.fit(X_train, y_train) # (This X_train is X_train_scaled; y_train
Out[136]:
In [137]:
# hyper parameters
gs.best_estimator_
Out[137]:
In [138]:
gs.best_score_
Out[138]:
Cross Validation¶
In [139]:
# Building a Support Vector Machine with best hyper parameters
svc_model = SVC(C=1, cache_size=200, class_weight=None, coef0=0.0,
decision_function_shape='ovr', degree=3, gamma='auto_deprecated',
kernel='linear', max_iter=-1, probability=False, random_state=None,
shrinking=True, tol=0.001, verbose=False)
svc_model.fit(X_train, y_train)
prediction = svc_model.predict(X_test)
In [140]:
# check the accuracy on the training set
print(svc_model.score(X_train, y_train))
print(svc_model.score(X_test, y_test))
In [141]:
print("Confusion Matrix:\n",confusion_matrix(prediction,y_test))
The SVM Model was applied using two approaches. It has been determined by Approach-1, programmatically, that out of two kernels (Linear / RBF), the best one would be RBF and the best score was close to 96%.
With the Approach-2, both the Linear and RBF were applied separately by splitting the data in 80/20 ration. In this approach, even the cross validation was implemented.